Effect of audiovisual perceptual training on the perception and production of consonants by Japanese learners of English

This study investigates whether L2 learners can be trained to make better use of phonetic information from visual cues in their perception of a novel phonemic contrast. It also evaluates the impact of audiovisual perceptual training on the learners pronunciation of a novel contrast. The use of visual cues for speech perception was evaluated for two English phonemic contrasts: the /v/–/b/–/p/ labial/labiodental contrast and /l/–/r/ contrast. In the first study, 39 Japanese learners of English were tested on their perception of the /v/–/b/–/p/ distinction in audio, visual and audiovisual modalities, and then undertook ten sessions of either auditory ( A training ) or audiovisual ( AV training ) perceptual training before being tested again. AV training was more effective than A training in improving the perception of the labial/ labiodental contrast. In a second study, 62 Japanese learners of English were tested on their perception of the /l/–/r/ contrast in audio, visual and audiovisual modalities, and then undertook ten sessions of perceptual training with either auditory stimuli ( A training ), natural audiovisual stimuli ( AV Natural training ) or audiovisual stimuli with a synthetic face synchronized to natural speech ( AV Synthetic training ). Perception of the /l/–/r/ contrast improved in all groups but learners trained audiovisually did not improve more than those trained auditorily. Auditory perception improved most for A training learners and performance in the lipreading alone condition improved most for natural AV training learners. The learners pronunciation of /l/–/r/ improved significantly following perceptual training, and a greater improvement was obtained for the AV Natural training group. This study shows that sensitivity to visual cues for non-native phonemic contrasts can be enhanced via audiovisual perceptual training. AV training is more effective 0167-6393/$ see front matter 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.specom.2005.04.007 q Portions of the work presented in this paper were presented at the 7th International Conference for Spoken Language Processing (16–20 September 2002, Denver, USA), at the International Congress of Phonetic Sciences (3–9 August 2003, Barcelona, Spain) and at the 8th International Conference for Spoken Language Processing (5–9 October 2004, Jeju island, South Korea). * Corresponding author. Tel.: +44 20 7679 7069; fax: +44 20 7383 4108. E-mail addresses: [email protected], [email protected] (V. Hazan), [email protected] (A. Sennema), [email protected] (M. Iba), [email protected] (A. Faulkner). 2 V. Hazan et al. / Speech Communication xxx (2005) xxx–xxx ARTICLE IN PRESS than A training when the visual cues to the phonemic contrast are sufficiently salient. Seeing the facial gestures of the talker also leads to a greater improvement in pronunciation, even for contrasts with relatively low visual salience. 2005 Elsevier B.V. All rights reserved.